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INTRODUCTION: 


Clinical  neuropsychologists  are  being  asked  to  make  statements  regarding  a  soldier’s  functional  skills,  ability  to 
return  to  active  duty,  and  competence  in  tasks  of  community  living.  These  concerns  are  what  are  referred  to  by 
the  term  ecological.  The  changing  referral  questions  bring  up  a  critical  issue:  most  neuropsychological  tests  are 
outdated  paper  and  pencil  administrations  that  were  developed  to  detect,  and  attempt  to  localize,  the  presence  of 
brain  damage  rather  than  to  be  ecologically  sound.  To  date,  research  establishing  the  ecological  validity  of 
neuropsychological  tests  has  been  limited.  Neuropsychological  tests  that  were  once  validated  for  their  ability  to 
detect  brain  damage  now  must  be  evaluated  for  their  utility  as  ecological  instruments.  Given  the  increasing 
prevalence  of  blast  injuries  to  the  head,  and  the  fact  that  many  brain  injuries  may  have  no  external  marker  of 
injury,  there  is  need  for  researching  innovative  assessment  methods  in  detecting  blast-related  brain  injury. 
Further,  since  such  injuries  can  greatly  interfere  with  the  soldier’s  ability  to  perform  complex  cognitive  and 
emotional  processing  tasks  involved  in  optimal  perfonnance  at  work  these  assessments  need  to  have  ecological 
validity.  Virtual  reality  (VR)  is  as  an  advanced  computer  interface  that  can  allow  a  soldier  to  become  immersed 
within  a  computer-generated  simulation.  Potential  VR  use  in  assessment  and  training  of  soldier  cognitive 
processes  is  becoming  recognized  as  technology  advances.  Since  VEs  allow  for  precise  presentation  and  control 
of  dynamic  perceptual  stimuli  (visual,  auditory,  olfactory,  gustatory,  ambulatory,  and  haptic  conditions),  they 
can  provide  ecologically  valid  (i.e.  military  relevant)  assessments  that  combine  the  veridical  control  and  rigor  of 
laboratory  measures  with  a  verisimilitude  that  reflects  real  life  situations.  Once  the  psychometric  properties  of 
these  norm-based  virtual  reality  measures  of  neuropsychological  functioning  are  established,  the  Army  would 
have  a  comprehensive,  flexible  and  scalable  VR  assessment  system  that  could  be  extended  to  investigate  a 
substantial  set  of  pragmatic  military  performance  assessment  questions. 


BODY: 

This  study  was  designed  as  an  initial  validation  of  the  VRCPAT  in  a  military  cohort.  The  University  of 
Southern  California  and  Madigan  Army  Medical  Center’s  National  Center  for  Telehealth  and  Technology  are 
collaborating  on  an  initial  pilot  study  to  validate  the  VRCPAT  with  active  duty  military  personnel.  We 
compared  paper-and-pencil  neuropsychological  tests,  ANAM,  and  VRCPAT  on  behavioral  measures.  While  our 
original  ending  date  for  the  Period  of  Performance  was  30  September  2011,  there  was  a  delay  in  protocol 
approval  that  led  to  delays  in  subject  recruitment.  We  requested  and  received  a  six  month  extension.  The  tasks 
that  remain  to  be  completed  include:  1)  Development  and  coding  of  the  data  from  the  VRCPAT,  ANAM,  and 
paper-and-pencil  analyses  unto  a  coherent  and  consistent  database  for  analyzing  the  data — including  data 
analytics  related  to  the  three  sets  of  measures  (i.e.,  VRCPAT,  ANAM,  and  paper-and-pencil)  for  psychometric 
validation  (Task  3);  and  2)  Preparation  of  data  for  publication  (Task  4).  Although  we  are  currently  furthering 
the  work  to  develop  all  aspects  of  the  data  base  (for  example,  incorporating  the  VRCPAT  Humvee  Stroop  and 
the  civilian  cohort),  we  have  completed  initial  analyses  on  the  VRPASAT  and  VRPVSAT.  Our  primary 
hypotheses  were:  1)  the  VRPASAT  (but  not  the  VRPVSAT)  would  be  significantly  correlated  with  the  paper- 
and  pencil  PAS  AT;  and  2)  the  VRPASAT  and  VRPVSAT  would  be  correlated  with  ANAM  subtests  assessing 
mathematical  processing  and  procedural  reaction  time. 

Participants  and  Procedure: 

A  total  of  49  (94%  were  male)  military  service  members  participated  in  the  study.  Fifty  percent  of  participants 
had  3  years  or  less  of  military  service  experience,  and  years  of  experience  ranged  from  1  to  24  years.  All 
participants  were  fluent  English  speakers  and  had  normal  or  corrected  to  normal  vision.  Each  participant  gave 
informed  consent  prior  to  participation  in  this  research  study.  After  informed  consent  was  obtained,  basic 
demographic  information  was  recorded.  Next,  participants  were  assessed  on  the  PASAT;  ANAM,  VRPASAT, 
and  VRPVSAT. 


Page  |  3 


Paper-and-Pencil  PAS  AT: 


Each  participant  was  presented  with  a  series  of  single  digit  numbers,  in  which  the  two  most  recent  digits  must 
be  summed.  For  example,  if  the  digits  “2”,  “6”  and  “3”  were  presented,  the  participant  was  to  respond  with  the 
correct  sums:  “8”  and  then  “9”.  Practice  trials  occurred  prior  to  the  first  trial.  Stimuli  are  presented  at  four 
different  rates  (2.4  s,  2.0  s,  1.6  s,  and  1.2  s).  The  duration  of  each  digit  was  approximately  .4  s. 

ANAM: 

The  following  ANAM  subtests  were  given  in  the  order  presented:  TBI  Questionnaire;  Sleepiness  Scale;  Mood 
Scale;  Simple  Reaction  Time;  Code  Substitution;  Procedural  Reaction  Time;  Math  Processing;  Matching  to 
Sample;  and  Go/No-Go. 

Middle  Eastern  City  (VRPASAT): 

While  virtually  ambulating  through  a  Virtual  Iraqi  City,  the  participant  was  presented  with  the  VRPASAT — a 
virtual  reality  variation  of  the  traditional  PASAT.  After  following  a  sergeant  to  6  zones,  the  participant  was 
tested  on  navigation  and  spatial  memory  by  being  instructed  to  turn  around  and  find  the  way  back  through  each 
of  the  zones  in  reverse  order. 

Middle  Eastern  Checkpoint  (VRPVSAT): 

An  area  of  the  Middle  Eastern  City  has  been  constructed  to  resemble  a  traffic  checkpoint  with  a  variety  of 
moving  vehicles  arriving,  stopping  and  then  moving  onward  into  the  city.  Participants  listened  to  a  virtual 
trainee  as  the  trainee  classified  passing  vehicles.  In  between  vehicle  presentations,  the  user  was  exposed  to 
visual  PASAT  stimuli.  Herein  the  numbers  are  presented  against  an  Iraqi  checkpoint  background. 

Data  Analytics 

Correlations  were  run  between  the  three  types  of  PASAT,  and  the  virtual  reality  based  PASATs  were  also 
correlated  with  various  tests  from  the  ANAM.  In  particular,  we  were  interested  in  the  relationship  of  the 
participants’  perfonnance  on  the  PASATs  and  the  mathematical  processing  and  procedural  reaction  time  tests. 

A  separate  set  of  analyses  were  conducted  to  assess  differences  in  participants’  perfonnance  on  the  various 
PASATs  employed.  A  3  (PASAT  type)  by  2  (session)  repeated  measures  ANOVA  was  utilized  to  serve  this 
purpose.  Separate  ANOVAs  were  employed  to  test  differences  between  the  virtual  reality  PASATs  in 
comparison  to  the  paper  and  pencil  version  during  the  first  two  sessions,  and  the  second  two  sessions. 

Results 

Correlations  between  performance  on  the  VRPVSAT  and  tests  in  the  ANAM  test  battery  revealed  that 
VRPVSAT  performance  is  conelated  with  mathematical  processing,  r  =  0.50,  p  <  0.001,  as  well  as  performance 
on  the  procedural  reaction  time  test,  r  =  0.41  ,P<  0.01.  Likewise,  the  VRPASAT  was  also  correlated  with  both 
mathematical  processing,  r  =  0.43,  p  <  0.01,  and  procedural  reaction  time  performance,  r  =  0.47,  p  <  0.01. 

The  correlation  analysis  involving  all  three  PASATs  revealed  that  the  VRPASAT  was  correlated  with  the  paper 
and  pencil  version  of  the  PASAT,  r  _,=  0.78,  p  <  0.001.  The  VRPVSAT  and  the  paper  and  pencil  PASAT  were 
not  significantly  correlated,  r  =  0.26,  p  =  0.07. 

The  ANOVA  involving  all  three  PASATs  and  using  the  first  two  sessions  of  the  paper  and  pencil  PASAT  (2.4 
and  2.0  respectively),  evidenced  a  main  effect  of  PASAT  type,  F(2,  47)  =  72.54,  p  <  0.001,  such  that  the  highest 
performance  was  found  during  the  VRPVSAT  as  a  paired-samples  t-tests  revealed  that  perfonnance  was 
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significantly  higher  than  perfonnance  on  both  the  VRPASAT,  t(48)  =  1 1.60,  p  <  0.001,  and  the  paper  and  pencil 
PASAT,  t(48)  =  3.43,  p  <  0.01.  The  VRPASAT  was  the  most  difficult  task  as  performance  was  significantly 
higher  on  the  paper  and  pencil  PASAT,  t(48)  =  11.12,  p  <  0.001.  A  session  main  effect  was  also  revealed,  as 
performance  on  the  first  session  was  higher  in  general,  F(l,  48)  =  39.84,  p  <  0.001.  The  interaction  between 
PASAT  type  and  session  was  also  significant,  F(2,  47)  =  10.76,  p  <  0.001,  due  to  the  lack  of  a  significant 
difference  in  performance  from  session  1  to  session  2  in  the  VRPVSAT.  The  ANOVA  utilizing  the  third  and 
fourth  sessions  of  the  paper  and  pencil  PASAT  (1.6  and  1.2  s  respectively)  evidenced  the  same  main  effect  and 
interaction  results.  However,  the  difference  between  perfonnance  on  the  paper  and  pencil  and  VRPASAT  was 
no  longer  significant,  t(48)  =  0.452,  p  =  0.65. 

Discussion 

The  Checkpoint  VRPVSAT  and  the  City  Walkthrough  VRPASAT  were  correlated  with  both  mathematical 
processing  and  procedural  reaction  time.  These  results  are  consistent  with  the  view  that  the  PASAT  measures 
speed  of  infonnation  processing;  mathematical  ability,  and  attention. 

Participant  performance  on  the  City  Walkthrough  VRPASAT  had  a  strong  positive  correlation  with 
performance  on  the  paper  and  pencil  PASAT,  demonstrating  high  levels  of  convergent  validity  between  the  two 
tests.  Differences  in  performance  between  the  two  PASATs  were  evidenced  during  the  first  two  sessions  of  the 
paper  and  pencil  PASAT,  but  not  during  the  second  two,  more  difficult,  sessions.  This  result  may  be  explained 
by  the  fact  that  the  City  Walkthrough  VRPASAT  includes  the  secondary  memory  task  as  well  as  the  added 
distracting  environmental  stimuli.  Thus,  even  when  the  interstimulus  interval  involved  is  longer  during  the 
virtual  reality  based  PASAT,  the  task  is  more  difficult  due  to  other  stimuli  requiring  cognitive  resources.  The 
City  Walkthrough  VRPASAT  may  also  allow  for  more  variability  in  performance  without  adding  to  participant 
frustration  by  speeding  up  the  test. 

As  hypothesized,  the  VRPVSAT  and  the  paper  and  pencil  PASAT  were  not  significantly  correlated.  These 
results  are  consistent  with  the  studies  using  various  patient  groups  (e.g.,  TBI;  and  multiple  sclerosis)  that  have 
found  PVSAT  perfonnance  to  be  superior  to  PASAT  performance.  This  superior  visual  performance  is  typically 
interpreted  as  representing  an  “interference”  framework,  in  which  the  articulation  of  the  response  aurally 
interferes  with  the  aural  presentation  of  the  digits.  Hence,  less  interference  exists  between  incoming  visual 
stimuli  and  the  articulated  response  on  the  VRPVSAT  than  on  the  VRPASAT.  As  a  result,  VRPVSAT  may 
proffer  a  more  robust  measure  of  speed  of  information  processing. 

Future  work  should  assess  the  clinical  validity  of  the  VRPASAT  and  VRPVSAT  in  a  clinical  population. 
Although  further  validation  with  an  intact  civilian  population  is  needed,  assessment  of  the  difference  between 
the  VRPASAT  and  VRPVSAT  may  have  important  clinical  implications. 

KEY  RESEARCH  ACCOMPLISHMENTS: 


Quarter  1: 

•  Meetings  with  Dr.  Gilliland’s  team  and  Dr.  Parsons’s  NeuroSim  Lab  at  USC/ICT.  Discussed  the  various 
stimuli  and  metrics  needed  for  an  executive  function  task  (e.g.,  Stroop)  being  embedded  in  a  virtual 
reality  environment  (e.g.,  Humvee)  with  safe  and  ambush  zones 

•  IRB  and  other  military  review  submitted 

•  Meeting  with  Dr.  Reger  at  Madigan  Army  Medical  Center.  He  is  currently  working  on  getting  a  cohort 
together  for  military  selection  of  participants 

•  Also  had  meeting  with  Dr.  Stetz  at  Tripler  Army  Medical  Center... focus  upon  extending  potential 
applicant  pool 

•  Developed  a  manual  and  have  trained  psychometrists  to  administer  and  score  VRCPAT 
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•  Identified  appropriate  programmer  to  make  changes  to  virtual  environment  per  discussions  with  C- 
SHOP  (Gilliland)  and  MAMC  (Reger) 

•  Working  with  programmer  to  make  changes  to  virtual  environment  per  discussions  with  C-SHOP 
(Gilliland)  and  MAMC  (Reger) 

•  Had  additional  meeting  and  send  release  to  Dr.  Stetz  at  Tripler  Army  Medical  Center... focus  upon 
extending  potential  applicant  pool 

•  Develop  some  data  output  resources  for  the  Virtual  Reality  Based  Stroop  task  and  potential  use  of 
psychophysiological  metrics. 

QUARTER  2: 

•  Meetings  with  Dr.  Gilliland’s  team  and  Dr.  Parsons’s  NeuroSim  Lab  at  USC/ICT. 

•  Discussed  the  data  analytics  necessary  for  scoring  and  analyzing  data  relative  to  various  stimuli  and 
metrics. 

•  Developed  scoring  rubrics  and  data  base  structure. 

•  Developed  codebook. 

QUARTER  3: 

•  Worked  with  Human  Subjects  Protection  Scientist  Sarah  L.  Donahue,  PhD,  MPH,  CIP  from  the  Human 
Research  Protection  Office  to  make  sure  data  transfer  from  Ft.  Lewis  Madigan  Army  Medical  Center 
met  U.S.  Army  Medical  Research  and  Materiel  Command  requirements 

•  Further  developed  and  programmed  the  scoring  for  the  VRCPAT  metrics  to  make  sure  that  they  comport 
well  with  C-SHOP  version  of  related  tests 

•  Worked  with  Ft.  Lewis/Madigan  Army  Medical  Center  to  discuss  the  preliminary  database  development 
and  the  necessary  aspects  of  de-identification. 

•  Developed  a  template  for  the  database  that  Ft.  Lewis/Madigan  Army  Medical  Center  will  populate  and 
send  to  TATRC 

•  Sent  an  example  database  to  Ft.  Lewis/Madigan  Army  Medical  Center  so  that  they  can  see  a  preferred 
approach  to  structuring  the  database 

QUARTER  4: 

•  Worked  with  Ft.  Lewis/Madigan  Army  Medical  Center  to  develop  the  database.  After  multiple 
iterations,  we  have  developed  a  database  that  represents  the  structure  necessary  for  data  analytics 

•  Received  data  from  Madigan  and  began  the  process  of  sorting  the  data  and  developing  the  appropriate 
scoring  algorithms 

•  Worked  with  C-SHOP  to  make  sure  that  we  had  all  the  data  necessary  for  cross  validation  of  the 
VRCPAT  with  ANAM—  Adjusted  data  to  be  better  analyzed  using  scoring  programs  (e.g.,  Matlab) 

•  Performed  a  number  of  scoring  conversions  and  conventions  to  the  data  base  to  make  sure  that  we  have 
both  primary  cognitive  score  metrics  and  constraint  analyses 

•  Performed  preliminary  analyses  comparing  the  virtual  reality-based  serial  assessment  tests  (VR  paced 
auditory  serial  assessment  test;  and  VR  paced  visual  serial  assessment  test)  with  standard  paper  and 
pencil  PAS  AT  and  various  ANAM  computerized  tests  (described  above). 

•  Note:  Worked  with  Mr.  Lance  Nowell  to  develop  and  implement  a  no  cost  extension.  Received  a  six 
month  “No  Cost  Extension”  (see  Appendix). 

REPORTABLE  OUTCOMES: 

•  Presented  “Virtual  Reality  in  Cognitive  Assessment  and  Return  to  Duty”  to  the  TATRC  sponsored 
Review  and  Analysis  Workshop:  Methods  and  Metrics  for  the  Biological  Assessment  of  Brain 
Dysfunction.  May  24-25,  2011.  Georgetown  Conference  Center.  Washington,  DC 

•  Manuscript  submitted  to  Medicine  Meets  Virtual  Reality 
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CONCLUSION: 


To  date  we  have  completed  many  of  the  milestones  set  forth  in  the  proposal:  1)  Hired  and  trained  Graduate 
Research  Assistants  (GRA)  in  administration  of  neuropsychological  measures  (Task  1)  and  recruited  subjects 
following  approval  of  human  subjects  committee  (Task  2).  Further,  throughout  the  project  we  consulted  with 
C-SHOP  on  proper  training  and  implementation  of  ANAM  for  validation  of  the  VRCPAT.  While  our  original 
ending  date  for  the  Period  of  Performance  was  30  September  2011,  there  was  a  delay  in  protocol  approval  that 
led  to  delays  in  subject  recruitment.  We  requested  and  received  a  6  month  extension.  The  tasks  that  remain  to  be 
completed  include:  1)  Development  and  coding  of  the  data  from  the  VRCPAT,  ANAM,  and  paper-and-pencil 
analyses  unto  a  coherent  and  consistent  database  for  analyzing  the  data — including  data  analytics  related  to  the 
three  sets  of  measures  (i.e.,  VRCPAT,  ANAM,  and  paper-and-pencil)  for  psychometric  validation  (Task  3).  2) 
Preparation  of  data  for  publication  (Task  4). 

Although  we  are  currently  furthering  the  work  to  develop  all  aspects  of  the  data  base  (for  example, 
incorporating  the  VRCPAT  Humvee  Stroop  and  the  civilian  cohort),  we  have  completed  initial  analyses  on  the 
VRPASAT  and  VRPVSAT.  Our  primary  hypotheses  were:  1)  the  VRPASAT  (but  not  the  VRPVSAT)  would  be 
significantly  correlated  with  the  paper-and  pencil  PASAT;  and  2)  the  VRPASAT  and  VRPVSAT  would  be 
correlated  with  ANAM  subtests  assessing  mathematical  processing  and  procedural  reaction  time.  The 
Checkpoint  VRPVSAT  and  the  City  Walkthrough  VRPASAT  were  correlated  with  both  mathematical 
processing  and  procedural  reaction  time.  These  results  are  consistent  with  the  view  that  the  PASAT  measures 
speed  of  infonnation  processing;  mathematical  ability,  and  attention. 

Participant  performance  on  the  City  Walkthrough  VRPASAT  had  a  strong  positive  correlation  with 
performance  on  the  paper  and  pencil  PASAT,  demonstrating  high  levels  of  convergent  validity  between  the  two 
tests.  Differences  in  performance  between  the  two  PASATs  were  evidenced  during  the  first  two  sessions  of  the 
paper  and  pencil  PASAT,  but  not  during  the  second  two,  more  difficult,  sessions.  This  result  may  be  explained 
by  the  fact  that  the  City  Walkthrough  VRPASAT  includes  the  secondary  memory  task  as  well  as  the  added 
distracting  environmental  stimuli.  Thus,  even  when  the  interstimulus  interval  involved  is  longer  during  the 
virtual  reality  based  PASAT,  the  task  is  more  difficult  due  to  other  stimuli  requiring  cognitive  resources.  The 
City  Walkthrough  VRPASAT  may  also  allow  for  more  variability  in  performance  without  adding  to  participant 
frustration  by  speeding  up  the  test. 

As  hypothesized,  the  VRPVSAT  and  the  paper  and  pencil  PASAT  were  not  significantly  correlated.  These 
results  are  consistent  with  the  studies  using  various  patient  groups  (e.g.,  TBI;  and  multiple  sclerosis)  that  have 
found  PVSAT  perfonnance  to  be  superior  to  PASAT  performance.  This  superior  visual  performance  is  typically 
interpreted  as  representing  an  “interference”  framework,  in  which  the  articulation  of  the  response  aurally 
interferes  with  the  aural  presentation  of  the  digits.  Hence,  less  interference  exists  between  incoming  visual 
stimuli  and  the  articulated  response  on  the  VRPVSAT  than  on  the  VRPASAT.  As  a  result,  VRPVSAT  may 
proffer  a  more  robust  measure  of  speed  of  information  processing. 

Future  work  should  assess  the  clinical  validity  of  the  VRPASAT  and  VRPVSAT  in  a  clinical  population. 
Although  further  validation  with  an  intact  civilian  population  is  needed,  assessment  of  the  difference  between 
the  VRPASAT  and  VRPVSAT  may  have  important  clinical  implications.  Over  the  next  six  months,  we  will  be 
continuing  our  efforts  at  developing  and  coding  the  data  from  the  VRCPAT,  ANAM,  and  paper-and-pencil 
analyses  unto  a  coherent  and  consistent  database  for  analyzing  the  data — including  data  analytics  related  to  the 
three  sets  of  measures  (i.e.,  VRCPAT,  ANAM,  and  paper-and-pencil)  for  psychometric  validation  (Task  3). 
Further,  during  the  next  several  months  we  will  be  preparing  manuscripts  for  publication. 
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